Upcoming Event: Babuška Forum

Differentiable Simulations and Interactive World Models

Krishna Kumar, Core Faculty, Oden Institute

Abstract

This talk introduces a framework for creating interactive, physically-realistic world models by merging advanced computer vision with differentiable physics simulations. Traditional approaches to building world models often force a trade-off between high visual fidelity and accurate physics. Our work bridges this gap by leveraging differentiable programming to build high-fidelity, interactive digital twins directly from real-world sensor data.

We demonstrate how modern computer vision techniques, including Neural Radiance Fields (NeRF) and 3D Gaussian Splatting, can reconstruct detailed 3D scenes from sparse video inputs. The core of our methodology is the integration of these reconstructions with fully differentiable physics engines, such as the Differentiable Material Point Method (DiffMPM). This novel combination enables us to solve challenging inverse problems: by comparing simulated outcomes with real-world observations, we use gradient-based optimization to automatically infer latent physical properties, such as soil friction angles and material stiffness, directly from video.

To overcome the computational demands of these physics solvers, we utilize Graph Neural Simulators (GNS) as a learned surrogate model that significantly accelerates simulation time. GNS represents physical systems as graphs of interacting particles, achieving speedups of over 100x compared to traditional methods while generalizing effectively to new scenarios. We also present a hybrid GNS-MPM approach that interleaves the learned simulator with the numerical solver to ensure long-term accuracy and conservation of physical laws.

The practical applications of this framework are highlighted through several key examples in robotics and autonomy. We present a system for generating Dynamic Costmaps from drone footage, where real-time 3D reconstruction and GNS-powered weather simulations predict terrain changes to enable safer navigation for autonomous vehicles. We also apply these methods to critical challenges in lunar exploration, such as simulating rover-regolith interaction to mitigate operational hazards. Ultimately, this research paves the way toward interactive "X2Sim" systems that can generate complex physical simulations from natural language or video prompts, opening a new paradigm for the design, control, and optimization of autonomous systems.

Biography

Dr. Krishna Kumar is a J. Neils Thompson Centennial Teaching Fellow and Associate Professor at the University of Texas at Austin. His research is at the intersection of AI/ML, numerical simulations, and robotics. He directs a $7M NSF-funded national ecosystem for AI integration in civil engineering and received an NSF CAREER Award in 2024. His research involves developing differentiable simulations, graph neural networks and control methods for autonomous construction and navigation in extreme environments. As an educator, he received the Dean's Award for Outstanding Teaching at UT Austin and runs coding clubs at Austin Public Libraries teaching AI/ML and robotics to children ages 7-12.